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Claims 

1 . (Currently amended) A method of recovering a three-dimensional scene from 
two-dimensional images, the method comprising: 

providing a sequence of frames; 

dividing the sequence of frames into frame segments wherein the frames in the 
sequence comprise feature points and wherein dividing the sequence of frames is divided into 
frame segments is based upon frames in each frame segment having at least a thr e shold 
minimum number of feature points being tracked to at least one base frame in the frame 
segment b e tw ee n th e fram e s of th e fram e s e gm e nts ; 

performing three-dimensional reconstruction individually for each frame segment 
derived by dividing the sequence of frames; and 

combining the three-dimensional reconstructed segments together to recover a three- 
dimensional scene for the sequence of images. 

2. (Previously Presented) The method of claim 1 wherein performing includes 
creating at least two virtual key frames for each of the segments, wherein the virtual key 
frames are only a subset of the images in a segment but are a representation of all of the images 
in that segment. 

3. (Canceled) 

4. (Original) The method of claim 1 wherein performing further includes: 
performing a two-frame structure-from-motion algorithm to create a plurality of local 

models for each segment; and 

combining the plurality of local models by eliminating scale ambiguity. 

5. (Original) The method of claim 4 further comprising: 
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bundle adjusting the combined local models to obtain a partial three-dimensional model 
for each segment; 

extracting virtual key frames from the partial three-dimensional model, wherein the 
virtual key frames include three-dimensional coordinates for the images and an associated 
uncertainty; and 

bundle adjusting all segments to obtain a complete three-dimensional model. 

6. (Original) The method of claim 1 further including: 
identifying feature points in the images; 

estimating three-dimensional coordinates of the feature points; and 
estimating a camera rotation and translation for a camera that captured the sequence of 
images. 

7. (Original) The method of claim 1 wherein combining includes performing a 
non-linear minimization process across the different segments through bundle adjustment. 

8. (Original) A computer-readable medium having computer-executable 
instructions for performing the method recited in claim 1 . 

9. (Currently Amended) A method of recovering a three-dimensional scene from 
two-dimensional images, the method comprising: 

identifying a sequence of two-dimensional frames that include two-dimensional images; 

dividing the sequence of frames into segments, wherein a segment includes a plurality 
of frames and wherein dividing includes, identifying a base frame, identifying feature points in 
the base frame; and determining the segments such that every frame in a segment has at least a 
predetermined percentage of feature points identified in the base frame ; 

for each segment, encoding the frames in the segment into at least two virtual frames 
that include a three-dimensional structure for the segment and an uncertainty associated with 
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the segment and wherein encoding includes choosing at least two frames in the segment that 
are at least a threshold number of frames apart; 

for each of the at least two chosen frames, projecting a plurality of three-dimensional 
points into a corresponding virtual frame; and 

for each of the at least two chosen frames, projecting an uncertainty into the 
corresponding virtual frame . 

10. (Canceled) 

1 1 . (Original) The method of claim 9 wherein the segments vary in length and 
wherein the length is associated with the number of frames in the segment. 

12. (Original) The method of claim 9 further including: 
identifying feature points in the sequence of two-dimensional frames; 
estimating three-dimensional coordinates for the feature points; and 
estimating camera rotation and translation for the feature points. 

13. (Original) The method of claim 12 wherein estimating the three-dimensional 
coordinates includes applying a two-frame structure-from-motion algorithm to the sequence of 
two-dimensional frames. 

14. (Original) The method of claim 9 further including: 
dividing a segment into multiple frame pairs; 

applying a two-frame structure-from-motion algorithm to the multiple frame pairs to 
create a plurality of local models; and 

scaling the local models so that they are on a similar coordinate system. 

15. (Previously Presented) The method of claim 14 wherein each of the multiple 
frame pairs includes a common base frame and one other frame in the segment. 
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16. (Original) The method of claim 15 further including interpolating frames 
between the multiple frame pairs. 

17. (Canceled) 

18. (Original) The method of claim 9 further including bundle adjusting the virtual 
frames from the segments to create a three-dimensional reconstruction. 

19. (Original) The method of claim 9 further including identifying feature points in 
the frames by using motion estimation. 

20. (Original) The method of claim 19 wherein the motion estimation includes: 
creating a template block in a first frame including a feature point and neighboring 

pixels adjacent the feature point; 

creating a search window used in a second frame; and 

comparing an intensity difference between the search window and the template block to 
locate the feature point in the second frame. 

21 . (Original) The method of claim 9 wherein at most two virtual frames are used. 

22. (Original) A computer-readable medium having computer-executable 
instructions for performing the method recited in claim 9. 

23. (Currently Amended) A method of recovering a three-dimensional scene from a 
sequence of two-dimensional frames, comprising: 

(a) identifying at least a first base frame in a sequence of two-dimensional frames; 

(b) adding the at least first base frame to create a first segment of frames of the 
sequence; 
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(c) id e ntifying selecting feature points in at least the first base frame in the first 
segment of frames in the sequence ; 

(d) analyzing a next frame in the sequence to identify the selected feature points in 
the next frame; 

(e) determining wh e th e r a threshold number of the selected feature points from the 
base frame that are also identified in the next frame; and 

(f) if a thr e shold the number of the selected feature points from the base frame that 
are also identified in the next frame is greater than or equal to a threshold number , adding the 
next frame to the first segment of frames of the sequence ^-and 

(g) r e p e ating (d) through (f) for subs e qu e nt fram e s until th e numb e r of f e atur e points 

in a fram e falls b e low th e thr e shold numb e r . 

24. (Currently amended) The method of claim 23 further including if the number of 
the selected feature points from the base frame that are also identified in the next frame is less 
than the threshold number, adding the next frame to a second segment of frames of the 
sequence and designating the next frame that falls below the threshold number as a second 
base frame in a second segment. 

25. (Original) The method of claim 23 further including performing motion 
estimation to identify the feature points. 

26. (Original) The method of claim 23 further including using corners as the feature 

points. 

27. (Original) The method of claim 23 wherein the number of frames comprising a 
segment varies between segments. 
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28. (Previously Presented) The method of claim 23 further including creating two 
virtual key frames per segment. 

29. (Original) The method of claim 28 further including bundle adjusting the virtual 
key frames of all the segments to obtain a three-dimensional reconstruction. 

30. (Original) A computer-readable medium having computer-executable 
instructions for performing the method recited in claim 23. 

3 1 . (Currently Amended) In a method of recovering a three-dimensional scene from 
a sequence of two-dimensional frames, an improvement comprising dividing a long sequence 
of frames into segments and reducing the number of frames in each segment by representing 
the segments using between two and five representative frames per segment, wherein the 
representative frames are used to recover the three-dimensional scene and remaining frames 
are discarded so that the three-dimensional scene is effectively compressed, wherein dividing 
the long sequence into segments includes identifying a base frame and tracking feature points 
between frames in the sequence and the base frame and ending a segment whenever a frame 
does not contain a predetermined threshold of feature points that are contained in the base 
frame . 

32. (Original) The method of claim 3 1 wherein each of the representative frames 
have an uncertainty associated therewith. 

33. (Original) The method of claim 31 wherein the long sequence includes over 75 
frames. 

34. (Original) The method of claim 3 1 wherein dividing the long sequence into 
segments includes identifying a base frame and tracking feature points between frames in the 
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sequence and the base frame and ending a segment whenever a frame does not contain a 
predetermined threshold of feature points that are contained in the base frame. 

35. (Original) The method of claim 3 1 further including performing a two-frame 
structure-from-motion algorithm on each of the segments to create a partial model. 

36. (Currently Amended) A computer-readable medium having computer- 
executable instructions for performing a method comprising: 

providing a sequence of two-dimensional frames; 
dividing the sequence into segments; 

calculating a partial model for each segment , wherein the partial model includes the 
same number of frames as the segment it represents and wherein the partial model that 
includes three-dimensional coordinates and camera pose , the camera pose comprising rotation 
and translation, for features within the frames; 

extracting virtual key frames from each partial model, the virtual key frames having 
three-dimensional coordinates for the frames and an uncertainty associated with the frames; 
and 

bundle adjusting the virtual key frames to obtain a complete three-dimensional 
reconstruction of the two-dimensional frames. 

37. (Previously presented) An apparatus for recovering a three-dimensional scene 
from a sequence of two-dimensional frames by segmenting the frames, comprising: 

means for capturing two-dimensional images; 
means for dividing the sequence into segments; 

means for calculating a partial model for each segment that includes three-dimensional 
coordinates and camera pose for features within the frames; 

means for extracting virtual key frames from each partial model; and 
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means for bundle adjusting the virtual key frames to obtain a complete three- 
dimensional reconstruction of the two-dimensional frames. 
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